04. Learning Rate

Learning Rate Tuning #1

Say you're training a model. If the output from the training process looks as shown below, what action would you take on the learning rate to improve the training?

Epoch 1, Batch 1, Training Error: 8.4181
Epoch 1, Batch 2, Training Error: 8.4177
Epoch 1, Batch 3, Training Error: 8.4177
Epoch 1, Batch 4, Training Error: 8.4173
Epoch 1, Batch 5, Training Error: 8.4169
SOLUTION: Increase the learning rate

Learning Rate Tuning #2

Say you're training a model. If the output from the training process looks as shown below, what action would you take on the learning rate to improve the training?

Epoch 1, Batch 1, Training Error: 8.71
Epoch 1, Batch 2, Training Error: 3.25
Epoch 1, Batch 3, Training Error: 4.93
Epoch 1, Batch 4, Training Error: 3.30
Epoch 1, Batch 5, Training Error: 4.82
SOLUTION:
  • Decrease the learning rate
  • Use an adaptive learning rate

Next Concept